Search for: All records

Photorealistic avatars have become essential for immersive applications in virtual reality (VR) and augmented reality (AR), enabling lifelike interactions in areas such as training simulations, telemedicine, and virtual collaboration. These avatars bridge the gap between the physical and digital worlds, improving the user experience through realistic human representation. However, existing avatar creation techniques face significant challenges, including high costs, long creation times, and limited utility in virtual applications. Manual methods, such as MetaHuman, require extensive time and expertise, while automatic approaches, such as NeRF-based pipelines often lack efficiency, detailed facial expression fidelity, and are unable to be rendered at a speed sufficent for real-time applications. By involving several cutting-edge modern techniques, we introduce an end-to-end 3D Gaussian Splatting (3DGS) avatar creation pipeline that leverages monocular video input to create a scalable and efficient photorealistic avatar directly compatible with the Unity game engine. Our pipeline incorporates a novel Gaussian splatting technique with customized preprocessing that enables the user of ”in the wild” monocular video capture, detailed facial expression reconstruction and embedding within a fully rigged avatar model. Additionally, we present a Unity-integrated Gaussian Splatting Avatar Editor, offering a user-friendly environment for VR/AR application development. Experimental results validate the effectiveness of our preprocessing pipeline in standardizing custom data for 3DGS training and demonstrate the versatility of Gaussian avatars in Unity, highlighting the scalability and practicality of our approach. 

Objective: Children and adolescents with intellectual and developmental disabilities (IDD), particularly those with autism spectrum disorder, are at increased risk of challenging behaviors such as self-injury, aggression, elopement, and property destruction. To mitigate these challenges, it is crucial to focus on early signs of distress that may lead to these behaviors. These early signs might not be visible to the human eye but could be detected by predictive machine learning (ML) models that utilizes real-time sensing. Current behavioral assessment practices lack such proactive predictive models. This study developed and pilot-tested real-time early agitation capture technology (REACT), a real-time multimodal ML model to detect early signs of distress, termed “agitations.” Integrating multimodal sensing, ML, and human expertise could make behavioral assessments for people with IDD safer and more efficient. Methods: We leveraged wearable technology to collect behavioral and physiological data from three children with IDD aged 6 to 9 years. The effectiveness of the REACT system was measured using F1 score, assessing its usefulness at the time of agitation to 20s prior. Results: The REACT system was able to detect agitations with an average F1 score of 78.69% at the time of agitation and 68.20% 20s prior. Conclusion: The findings support the use of the REACT model for real-time, proactive detection of agitations in children with IDD. This approach not only improves the accuracy of detecting distress signals that are imperceptible to the human eye but also increases the window for timely intervention before behavioral escalation, thereby enhancing safety, well-being, and inclusion for this vulnerable population. We believe that such technological support system will enhance user autonomy, self-advocacy, and self-determination. 

Abstract This study explores the intersection of Theory of Mind (ToM) abilities and driving performance among novice drivers, with a focus on autistic individuals. The purpose is to investigate how ToM deficits may impact driving behaviors and decision-making, ultimately informing the development of tailored interventions and training programs for autistic drivers. We conducted a series of driving simulations using a custom-built driving simulator, capturing multimodal data including driving performance metrics, attention allocation, and physiological responses. Participants were categorized based on NEPSY scores, which assess ToM abilities, and self-reported autism spectrum disorder (ASD) diagnosis. Driving tasks were designed to simulate real-world scenarios, particularly focusing on intersections and merging, where ToM skills are crucial for safe navigation. Our analysis revealed differences in driving behaviors among participants with varying ToM abilities as determined through the NEPSY. Participants with lower NEPSY scores exhibited less smooth driving behaviors, increased risk-taking tendencies, and differences in attention allocation compared to those with higher scores. Alternatively, individuals with ASD displayed comparable driving patterns overall. ToM abilities influence driving behaviors and decision-making, particularly in complex social driving scenarios. Tailored interventions addressing ToM deficits and stress management could improve driving safety and accessibility for autistic individuals. This study underscores the importance of considering social cognitive factors in driving education and licensure pathways, aiming for greater inclusivity and accessibility in transportation systems. 

Computer-based job interview training, including virtual reality (VR) simulations, have gained popularity in recent years to support and aid autistic individuals, who face significant challenges and barriers in finding and maintaining employment. Although popular, these training systems often fail to resemble the complexity and dynamism of the employment interview, as the dialogue management for the virtual conversation agent either relies on choosing from a menu of prespecified answers, or dialogue processing is based on keyword extraction from the transcribed speech of the interviewee, which depends on the interview script. We address this limitation through automated dialogue act classification via transfer learning. This allows for recognizing intent from user speech, independent of the domain of the interview. We also redress the lack of training data for a domain general job interview dialogue act classifier by providing an original dataset with responses to interview questions within a virtual job interview platform from 22 autistic participants. Participants’ responses to a customized interview script were transcribed to text and annotated according to a custom 13-class dialogue act scheme. The best classifier was a fine-tuned bidirectional encoder representations from transformers (BERT) model, with an f1-score of 87%.